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OVERCOMING THE CHALLENGE OF LOW 
RESPONSE RATE IN ALUMNI SURVEYS 

Abstract 



Since the mid-1980s our institution has developed an alumni survey as part of its student 
tracking system. From the beginning, the alumni survQ^ was the most challenging component of that 
research. In spite of our efforts, we are still struggling to have an efficient mechanism that would give 
us accurate information about our former students’ college experience, their assessment of the 
institution and of their academic preparation. A major challenge in this effort has been the low 
response rate. 

We want to share with the audience the combination of strategies we have been implementing 
to overcome the limitations of the study, the results we have achieved so far, and our next steps to 
improve this research. 
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OVERCOMING THE CHALLENGE OF LOW 
RESPONSE RATE IN ALUMNI SURVEYS 

As in many American colleges, assessment was an incipient activity at the University of 
Puerto Rico in the early 1980s. Our planning oflBces were incorporating institutional research 
functions on a small scale, mainly as a random, unaiticulated activity to document some areas of the 
academic life; such as; program demand, job opportunities for our majors, and institutional statistics. 
But, as Terenzini (Stark, 1994, p. 523) points out, at this stage many of those efforts were not 
coordinated in any way and were not part of any comprehensive, institutional plan for ongoing 
systematic self-study and improvement. 

With the switch to strategic planning, the need for systematic collection of information to 
document the institution’s competitive advantage over the others became evident. George Keller 
(1983) had stated: “Before any college or university executives begin to shape an academic strategy, 
they need to gather the proper information on which to base the strat^. . . Information, quality, and 
people: these are the critical items for strategic planning to be effective.”* Keller acknowledged that 
most institutions lacked information on student outcomes and referred to the work of Alexander 
Astin, in UCLA, as a pioneering effort in providing information about the student experience in 
College. The works of Astin (1977), Lenning (1977, 1980), Bowen (1980), Ewell (1983, 1984), 
among others, influenced the early stages of data collection in Humacao. 

Our first systematic institutional research project was the Student Tracking System initiated 
in 1984-85. Under the leadership of the Central Administration OflBce, the Systemic Committee for 
Institutional Studies gathered the representatives from the eleven (11) units of our university System. 
Collaboratively, the group developed and pilot-tested the different instruments that would comprise 
the Humacao Student Tracking System. 
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Figure 1 : Student Tracking System at the University in Humacao 



The main goal of this longitudinal research project was to provide an objective mechanism 
to measure the students' expectations, achievements, and difiScuhies throughout their experience on 
the institution. We wanted to find out who entered the college, what they expected from it, and how 
well we helped them achieve their expectations. This would become our institutional assessment 
project and eventually one of the elements of our institutional effectiveness and oirtcomes assessment 
project. 



An important component of this project was the alumni survey. From the beginning, this was 
the most challenging aspect of our research. In spite of our efforts, we were still struggling to have 
an efficient mechanism that would give us accurate information about our former students’ college 
experience as well as their assessment of the institution and of their academic preparation. 

This paper presents the combination of strategies we have been implementing from 1996-97 
to the present to overcome the limitations of our alumni study, the results we have achieved so far, 
and our next steps to improve this research. 



The Alumni Study 



Historical Background 



Humacao began the Student Tracking System in Augustl984 and by June 1987 we were 
mailing questionnaires to our associate degree graduates. Until 1991 we surveyed groups both one- 
year and three-years after graduation. With the first groups we did one mailing with two or three 
follow-ups, only to those students who were part of the track we were following fi’om the fi'eshmen 
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years. This accounted for initial high response rates ranging from 58 to 74 percent. But when the 
number of graduates increased our response rates fell dramatically below 20%. Eventually, in 1991- 
92, we stopped our alumni research. 

For a period of almost eight years, we relied on two other sources of information on student 
outcomes; the graduating class questionnaire performed at institutional level, and the alumni studies 
prepared by some of the academic departments. However, there were a few problems with this 
strategy. 

Despite the fact that we had in place a good graduation instrument for our seniors, we felt 
that we were missing valuable information concerning our graduates’ performance in their jobs and 
at graduate level. Our seniors gave us good feedback in terms of their experience throughout their 
college years, valuable assessment of student support services and of the College environment, and 
some general information in terms of their satisfaction with their programs, their faculty, and the 
quality of the teaching-learning process. But we did not hear from them beyond the graduation point. 

Some of our programs with external accreditation nianaged to survey employers and their 
own alumni to find out whether they were employed or studying at graduate level. But this was on 
a very small scale and only for accreditation purposes. 

In 1994, our Chancellor and the institutional researcher attended James Nichols’ workshop 
“Institutional Effectiveness and Student Outcomes Assessment”, in Nashville, Tennessee.^ The 
Chancellor decided to adopt the institutional effectiveness paradigm which focuses on assessing 
student outcomes as a means to document the achievement of academic and service objectives as well 
as broader institutional goals and mission. An important component of that assessment should be the 
feedback of graduates, their employers as well as the graduate schools that admit them for graduate 
work. 



A third reason for reinitiating our alumni study was our college accreditation. Humacao 
would have the visit of the Middle States Association of Colleges and Schools in the Spring of 1999 
(due to Hurricane Georges it was postponed for the Fall of 1999). From our previous accreditation 
in 1988-89, we knew that the Middle States Association includes outcomes iu^sessiiiciit as the core 
of the accreditation process. This time they emphasized student outcomes even more (1991, 1994, 
1996, 1998). Thus in 1996-97 we reinitiated our alumni study. 



Methodology 



We had learned a couple of lessons from our previous experience and from the experience of 
other researchers with this same topic. 
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Lesson #1: The Alumni Study had to be a collaborative effort between the faculty in the 
academic programs and the institutional researchers . The first decision we made was to include 
academic departments in this effort in order to collect both the institutional assessment and the 
program assessment. We organized the Institutional Committee for the Alumni Study (See Figure 2) 
with a representative fi'om each academic department. The representatives were either the Program 
Coordinators, the Academic Advisors, or a faculty member designated by the Department Chair. 





Figure 2: Institutional Conmiittee for the Alumni Study 

As part of this effort, the Assessment Program hired one Faculty member who would share 
coordinating responsibilities with the Assistant Researcher from the Institutional Research Division. 
These two coordinators would be in charge of providing the leadership and the technical assistance 
the departments needed to initiate the study. 

Specifically, the Alumni Committee would; 

1. Review previous alumni questionnaires used at institutional and department level 

2. Produce a revised instrument which would collect both the data needed by the Institution 
and by each department, thus the information could be grouped in one institutional profile and 
segregated by academic department. 



3. Prepare the alumni directory 
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4. Mail the questionnaire 

5. Prepare the Alumni Profile for each department 

Lesson #2: Graduates’ lovaltv is to their faculty and their programs. We had witnessed that 
graduates responded feirly well to the surveys sent by their specific programs, especially those which 
received external accreditation. Social Work, Nursing (associate and bachelor’s degrees). Physical 
Therapy and Occupational Therapy conducted their alumni and employers surveys on a regular basis 
with response rates over the 40%. 

We realized that this was one of the major limitations of our previous study. Our graduates 
could not relate to us, the institutional researchers, as well as they did to their professors. Thus, we 
would seek the assistance of the Program Coordinators in the Committee for sending, following up, 
and collecting questionnaires. 

Besides knowledge gained from experience, there were two other factors that promised to 
help us to be successful in this new attempt. First of all, there were three of us sharing the research 
responsibility instead of one single researcher. Second, we were all more experienced researchers. 
This would allow us to try out other research methodologies besides the mail-in survey. 

This second cycle of the Alumni Study includes Humacao University College graduates fi-om 
1995, 1993/1996, and 1994/1997. 



The 1995 Alumni Study includes a random sample of 100 graduates stratified by academic 
programs who, by 1996-97, had completed one year away fi^om the College. Those graduates 
represent 17% of their class. We mailed them the Alumni Questionnaire that includes twenty-two 
questions. There was one additional follow up. 



The 1993/1996 Alumni Study was a little more ambitious. Instead of one group we decided 
to ^proach two classes: 1993 graduates who had been away for three years and 1996 graduates who 
had been out for one year. Instead of samples, we sent the survey to the universe of graduates: the 
593 for the 1993 class and 671 for the 1996. Besides the survey we wanted to conduct focus groups. 
The members of the Institutional Coiiuriittee mailed the questionnaires and helped us with their 
follow-up and collection. We sent the questionnaire to the graduates fi^om the three departments 
which had not developed the program assessment section. Graduates received a follow-up by 
telephone. 



The 1994/1997 Alumni Study is the last one in this second cycle. Again we surveyed one 
year and three year graduates. Our conclusion after analyzing the findings for the other two studies 
was that we could get about the same results with a random, stratified sample. Thus, for 1994/97 we 
chose a random sample stratified by academic programs with a margin error of .03% and a 
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confidence level of 99%. We used Cheby ShefiPs theorem for calculating the sample. Out of 13 
academic departments, 12 (92%) prepared the program assessment section. The Coordinators of the 
Study mailed the questionnaires. 

We also conducted focus groups. From the sample for the survey of each class, we chose 
another sample for the focus groups. From the 1994 class we chose 25 graduates; from the 1997 
class we chose 30. The Assistant Researcher contacted each graduate by telephone and confirmed 
his/her attendance to the group interviews. He mailed the Alumni Questionnaire and a letter 
reminding them of the date for the focus group. 

We also did a Graduate Schools and Employer Study to complement the Alumni Study. It 
used personal interviews as research technique. We interviewed a random sample of employers and 
a sample group of graduate schools. To prepare the Employers Directory we used the telephone 
guide. In the phone book for the Humacao Region we identified 4,298 possible employers. We 
stratified the universe by economic sector. Then we chose a random stratified sample of 405 
employers with .05% margin of error and 95% level of confidence. Since we wanted to use personal 
interviews as research technique, we decided to interview only a 5% of that sample. Randomly we 
picked 24 companies for the interviews. Each company was contacted by telephone and the 
appointment for the interview was set. 

To make the Graduate Schools directory, we interviewed Department Chairs and faculty 
members from each program in our College which kept contact with their graduates. We prepared 
a list of the graduate schools our former students attended. At this point we were running out of time 
and decided to work with a small group of them just to have an idea about how they perceived our 
graduates. Out of 86 graduate schools identified, we contacted 5 on the phone. As with employers, 
we set an appointment for the interviews. 

Results ; 

Alumni Questionnaire . Table 1 presents the response data for the three mail-in surveys. 
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Table 1 

Distribution of Response for Alumni Studies 1995. 1993/1996. and 1994/1997 



Graduation Class 


Total of 
Graduates 


Sample 


Total of 
Response 


% 


1995 (1 year) 


594 


100 


28 


28 


1996 (1 year) 


671 


— 


136 


20 


1997 (1 year) 


617 


259 


106 


41 


1993 (3 years) 


593 


— 


70 


12 


1994 (3 years) 


615 


259 


87 


33 



The results of the 1995 Alumni Study were unexpected. Out of 13 departments, only 6 (46%) 
developed the program assessment section. Therefore, most of the graduates only received the 
institutional assessment survey. Only 28% out of a sample of 100 graduates completed the 
questioimaire (see Table 1). We worked for a higher response rate. But we were not discouraged 
because we had decided to try new ways to improve the response. 

In 1997-98, we tried to accomplish the following goals: 

1 . increase the number of programs participating in the study 

2. increase the response rate 

3. survey two groups simultaneously: former students who had been away after 
graduation for one year and for three years 

We accomplished two of the three goals proposed. Out of 13 academic departments, 10 
developed the program assessment section. This represents 4 more than the previous year. We also 
accomplished the study of two cohorts of graduates simultaneously. The response, however, did not 
increase. There was a 12% of response for 1993 graduates and a 20% for those of 1996. Compared 
to the previous study of one-year graduates, the percentage decreased by 8%. 

The addition of more departments in the study required a greater efibrt to respond to the 
needs of each program in the stage of constructing the instruments. Therefore we postponed the 
focus groups. 
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For 1994/1997 we worked with samples. As Table 1 shows, the response of one-year 
graduates increased to 41%; this is 21% higher than the response for the 1996 group. The response 
for three-year graduates was 33%. This was 21% higher than that for the 1 993 graduates. 

In terms of the questionnaire, the first question asked the graduates to identify the goals that 
were important for them while studying at the HUC and to identify those that they accomplished as 
a result of studying there. Table 2 shows that there were five (5) goals identified both as expected 
and achieved by more than 50% of most of the graduates: 

• prepare for a profession 

• increase knowledge and understanding of an academic field 

• receive a degree or certificate 

• improve knowledge, technical skills, or competencies for the job or career 

• formulate plans or goals 

Those goals can be considered academic goals. However, the greatest gains occurred in goals 
that relate more to personal development rather than academic skills, and which were not expected 
by a high number of graduates. These were: 

• develop the ability to be independent, self-sufficient , and adaptable 

• increase self-confidence 

• learn skills to enrich their lives or to make them a better person 

• improve leadership skills 
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Percentage Distribution of 1993-1997 Alumni bv Expected/ Achieved Goals (50% or More of the Graduates') 
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Important Goals 


1 . Prepare for a profession. 


2. Increase knowledge and understanding 
of an academic field. 


3. Receive a certificate or degree. 


4. Develop the ability to be independent, 
self-sufficient, and adaptable. 


5. Formulate professional plans or goals. 


6. Improve knowledge, technical skills or 
competencies for the job or career. 


7. Meet people and improve their ability to 
relate to others. 


8. Increase self-confidence. 


9. Learn skills to enrich their lives or to 
make them a better person. 


10. Improve leadership skills. 


1 1 . Increase opportunities for promotion or 
salary raise. 
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Graduates evaluated their academic experience at the HUC. As Table 3 shows, th^ indicated 
that they developed the following skills a lot: 

• work independently 

• follow directions 

• work collaboratively in groups 

• be persistent with difficult tasks 

• define, solve problems, and make decisions 

Graduates consider that they developed the following skills somewhat; 

• understand and apply computer literacy concepts 

• understand and appreciate the arts 

• understand and apply scientific principles and methods 

• take care of their physical and mental health 

• understand different philosophies and cultures 

• speak effectively in English 
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Table 3 



Alumni Evaluation of their Academic Experience at the HUC. Alumni from 1993 to 1997 





YEARS 


SKILLS DEVELOPED AT THE HUC 


1993 


1994 


1995 


1996 


1997 


Write effectively in Spanish 


3.5 


3.4 


3.4 


3.4 


3.5 


Write effectively in English 


3,1 


3,1 


3,1 


3.0 


3.1 


Speak effectively in Spanish 


3,5 


3,5 


3.3 


3.4 


3.5 


Speak effectively in English 


2,9 


2,9 


2.9 


2.9 


3.0 


Comprdiension of written material in Spanish 


3.5 


3.5 


3.3 


3.5 


3.5 


Comprehension of written material in English 


3.1 


3.2 


3.1 


3.2 


3.2 


Woik ind^endently 


3.7 


3.6 


3.5 


3.6 


3.6 


Manage personal and family finances 


3.2 


3.3 


3.1 


3.0 


3.0 


Self'leaming 


3.3 


3.3 


3.5 


3.2 


3.1 


Understand graphic infonnation 


2.9 


2.9 


2.9 


3.2 


3.0 


Use Library resources 


3.3 


3.2 


3.4 


3.4 


3.3 


Follow directions 


3.6 


3.6 


3.5 


3.6 


3.6 


Understand issues that affect you as consumer 


3,1 


3.1 


2.8 


3.1 


3.1 


Take care of your physical and mental heahh 


2.9 


2.9 


3.0 


2.9 


2.9 


Work collaboratively in groups 


3.5 


3.4 


3.6 


3.7 


3.7 


Manage time effectively 


3.4 


3.4 


3.6 


3.4 


3.3 


Be aware of your rights, duties, and privileges as citizen 


3.4 


3.4 


3.3 


3.3 


3.4 


Understand and apply mathematics in your daily life 


3.1 


3.1 


3.0 


3.1 


3.0 


Understand different philosophies and cultures 


2.9 


2.9 


2.8 


3.0 


3.0 


Be persistent with difficuh tasks 


3.6 


3.6 


3.4 


3.6 


3.6 


Define and solve problems and make decisions 


3.6 


3.6 


3.4 


3.7 


3.7 


Understand man’s interaction with his environment 


3.1 


3.1 


2.8 


3.2 


3.2 


Direct and lead others 


3.3 


3.3 


3.3 


3.3 


3.4 


Identify hypotheses, make inferences and draw correct 
conclusions 


3.2 


3.2 


3.3 


3.3 


3.3 


Understand and appreciate the arts 


2.8 


2.8 


2,7 


2.8 


2.8 


Understand and apply scientific principles and method 


2.7 


2.7 


3.0 


3.0 


2.8 


Understand and apply computer literacy concepts 


2.4 


2.5 


- 


2.8 


2.0 




N = 70 


N = 87 


N = 28 


N = 136 


N = 106 



A lot (4), somewhat (3}y very little (2), in no degree (1). 



Graduates assessed most of the different College services and its environment as good. They 
rated as exceUent the quality and the content of courses in their majors (3.6). They rated as average 
the parking facilities (2.0), career placement services (2.1), and athletic facilities (2.4). Table 4 
presents these ratings. 
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Table 4 



Alumni Evaluation of the HUC and its Services. Alumni from 1993 to 1997 



COLLEGE SERVICES AND 


YEARS 


ENVIRONMENT 


1993 


1994 


1995 


1996 


1997 


Teaching Process 

Teaching quality in general education courses 


3.2 


3.2 


3.5 


3.3 


3.2 


Teaching quality in courses of your major 


3.6 


3.5 


- 


3.6 


3.5 


Content of general education courses 


3.2 


3.2 


3.2 


3.2 


3.2 


Content of courses in your major 


3.5 


3.4 


3.8 


3.6 


3.6 


Grading system 


3.2 


3.2 


3.2 


3.2 


3.3 


Text 


3.0 


3.0 


3.0 


3.2 


3.2 


Exams, evaluation means, etc. 


3.1 


3.1 


3.1 


3.0 


3.1 


Faculty 

Dedication of professors in general education courses 


3.2 


3.1 


3.2 


3.2 


3.2 


Dedication of professors in your major 


3.5 


3.4 


3.6 


3.5 


3.5 


Teaching methods 


3.2 


3.1 


3.2 


3.2 


3.2 


Academic counseling 


3.0 


3.0 


2.9 


3.1 


3.1 


Attitude of professors toward students 


3.1 


3.1 


2.9 


3.0 


3.2 


Services 












Admission process 


3.1 


3.1 


3.0 


3.1 


3.0 


Registration process 


2.5 


2.5 


2.4 


2.6 


2.6 


Cashier 


2.8 


2.7 


2.8 


2.6 


2.6 


Student Aid 


2.9 


2.9 


3.0 


2.8 


2.8 


Counseling 


2.7 


2.7 


2.9 


2.7 


2.7 


Career Placement 


2.2 


2.1 


1.8 


2.2 


2.1 


Medical Services 


2.7 


2.7 


2.5 


2.5 


2.5 


Library 


3.0 


2.9 


2.9 


2.9 


2.9 


Computer Center 


2.7 


2.7 


3.3 


2.7 


2.8 


Cafeteria. 


2.6 


2.6 


2.7 


2.6 


2.7 


Social and cultural activities 


2.9 


2.9 


3.1 


2.9 


2.9 


Athletic Activities 


2.7 


2.6 


2.5 


2.6 


2.6 


Summer session 


2.5 


2.5 


2.4 


2.6 


2.6 


'Siuucut services m general 


2.8 


2.7 


2.8 


2.8 


2.8 


Physical Activities 












Classrooms 


2.6 


2.6 


2.7 


2.6 


2.5 


Theater 


3.0 


3.0 


2.9 


2.7 


2.7 


Auditoriums 


2.7 


2.7 


2.6 


2.5 


2.5 


Equipment and teaching laboratories 


2.6 


2.5 


2.7 


2.7 


2.6 


Equipment and computing laboratories 


2.6 


2.6 


2.9 


2.9 


2.8 


Atletic facilities 


2.6 


2.5 


2.2 


2.4 


2.4 


Parking facilities 


2.1 


2.1 


2.0 


1.8 


1.8 


Facilities for the handicapped 


2.5 


2.5 


2.9 


2.5 


2.5 


Physical facilities in general 


2.7 


2.7 


2.6 


2.6 


2.6 




N = 70 


N = 87 


N = 28 


N= 136 


N= 106 



Excellent (4), good (i), regular (Z), poor (1). 
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In general, graduates indicate that they are satisfied with the HUC. They also indicate that 
stud 3 ^g here definitely improved the quality of their lives. Table 5 illustrates these findings. 



Table 5 



Alumni Evaluation of the HUC 1993 to 1997 



EVALUATION OF 
HUC AND ITS SERVICES 


YEARS 


1993 


1994 


1995 


1996 


1997 


Level of satisfaction with the Humacao College 
[Very satisfied (5), satisfied (4), neutral (3), dissatisfied (2), very 
unsatisfied (J)] 


4.3 


4.2 


4.2 


4.4 


4.4 


Studying at the Humacao College improved the quahty of 
their lives. 

[Definitely (5), probably (4), undecided (3), probably not (2), definitely^ 
no (1)1 


4.7 


4.5 


4.6 


4.7 


4.7 



N=70 N=87 N=28 N=136 N=106 



Focus Groups . To improve the quality of the response we conducted three (3) focus 
groups; one session with 1994 graduates and two sessions with 1997 graduates. Table 6 presents 
the participation of the graduates in the focus groups. 

Table 6 



Distribution of Attendance to Focus Groups 1994-1997 Alumni 



Graduation Class 


Sample 


Attendants 


% 


1994 


25 


4 


16 


1997 


30 


12 


40 



Main findings for the focus groups; 

• Out of 1 6, 11 (69%) indicate that they still like their professions afl;er the job experience 

• 10 (62%) are employed in their fields 

• 10 (62%) received additional training for the jobs they were hired for 

• 15 (94%) were unsatisfied with their salaries 
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Even though we expected a higher attendance to the focus groups, we were pleased with the 
quality of the response from our graduates. As an example, we think that some of their responses 
to the first question, “Do you still like your profession after your experience at work? "are highly 
relevant: 



Industrial Chemist; 1 like it, but it is not what I expected. I think that any 
trained person can do the job”. 

Occupational Ther^ist; “I like it, but I am not employed in my field Other 
graduates are working as Occupational Thercpy Assistants, even though they do not 
have the credentials. They do the same job under a different title such as 
Recreational Therapist.” 

Social Worker: T like it, but there is a false expectation about what we would 
in the agencies [where we work]. They tell us [in the program] that we cae agents 
of change, but we are not The agencies want to clumge us violating our 
professional ethics. People who do not have the professional training do our work 
An example are custodians in the Courts of Justice. On the other hand some 
graduates have simple jobs that any high school graduate can do. ” 

Biologist; “I like it I think it gave me the basic tools in the science area. I 
think they should revise and update it, [the Program] specifically with more 
computers... ” 

Secretary: T am not working as a secretary, but as office coordinator in a 
private company. I think the professors [in the department of Office Systems 
Administration] are excellent I suggest that they update the curriculum. 

Besides this, 1994/1997 graduates provided useful information in terms of aspects of their 
majors that could be improved, courses that should be added, eliminated or updated, and some input 
in terms of what the future evolution of their profession would be according to their job experience. 

Employers Interviews . We interviewed only 21 employers. To set the interviews we made 
280 telephone calls or 13.3 calls for each interview. Table 7 presents the employers’ ratings of the 
general performance of our graduates; 
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Table 7 

Em ployers’ Rating of HUC Graduates’ Performance 



AREAS ASSESSED 


RATINGS 


L 


Attendance 


3.8 


2. 


Punctuality 


3.7 


3. 


Initiative 


3.8 


4. 


Ability to follow directions 


3.7 


5. 


Team-work ability 


3.8 


6. 


Receptive to supervision 


3.9 


7. 


Receptive to in-service training 


3.8 


8. 


Receptive to professional improvement and training 


3.8 


9. 


Interpersonal relations 


3.7 


10. 


Respect for safety rules 


3.6 


11. 


Respect for general organizational rules 


3.7 


12. 


General attitude toward the job 


3.8 


13. 


Technical knowledge of job 


3.8 


14. 


Expertise in the use of equipment required for the job 


3.8 


15. 


Verbal communication in Spanish 


3.8 


16. 


Written communication in Spanish 


3.6 


17. 


Verbal communication in English 


2.7 


18. 


Written communication in English 


2.6 


19. 


Quantity of work produced 


3.7 


20. 


Quality of work produced 


3.8 



• Over 80% of the employers interviewed rated our graduates as good/very good in almost all the 
areas we were assessing. 

• Between 10 and 38 percent rated them as average/poor in the following areas: 

follow safety rules (10%) 

Verbal communication in English (30%) 
written communication in English (3 8%) 

• Fifteen (15) 71% of the employers interviewed rated as very high their level of satisfaction with 
the job performance of our graduates; six (6) 29% rated it as high. 
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The employers provided us with a list of strengths and weaknesses of our graduates as 
employees in their organizations. They also provided specific recommendations to improve our 
programs. This is probably the part that will be most useful for the curricular revisions of our 
academic programs. 

Graduate Schools . Out of 86 graduate schools, we visited 5 (6%). In some of them we 
interviewed the director of the graduate program and in others we interviewed a staff member who 
was responsible for admissions and registration of the graduate candidate. 

The major findings for this component were: 

One (1) out of five (5) told us the amount of students they admitted fi"om our 
College. 

One (1) told us how our students compared with other graduate students in 
terms of their preparation. 

Two (2) told us the amount of our graduates who completed a graduate 
program. 

Three (3) of them rated our graduates as very good/good in many of the areas 
assessed in our instrument. 

Three (3) rated our graduates as average/poor in the following areas: written 
communication skills in English, research skills, and ability to work in-groups and 
independently. 

Three (3) of them indicated a high level of satisfaction with the performance 
of our graduates in the graduate courses. 

Like employers, the personnel of the graduate schools identified strengths and weaknesses 
of our graduates for conducting graduate studies. They also provided specific reccmmcndaticns to 
improve the academic background of our students. Those recommendations will be sent to our 
programs for their attention. 
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We are quite pleased with the results and the information collected with this second <ycle of 
our Alumni Study. We managed to increase both the quantity and the quality of the response. We 
can point out the following major accomplishments; 

1 . Successfiil combination of research techniques; 

• Questionnaire; Institutional and Program Assessment 

• Focus groups 

• Personal interviews 

2. Graduates response increased; 

• One-year graduates 

1995 - 28% 

1997 - 41% 

• Three-year graduates 

1993 - 12% 

1994 - 33% 

3 . Faculty participation increased ; 

« 1995 - 6 departments (46%) 

• 1996 - 10 departments (77%) 

• 1997 - 12 departments (92%) 

4. First institutional Study of Employers and Graduate Schools 

In general terms, the response of our graduates, their employers/supervisors, and 
professors/staff at graduate level shows that they perform at a satisfactory level. Employers, 
supervisors, and professors point out that we could improve their preparation in terms of verbal and 
written communication skills both in EngUsh and Spanish, critical analysis, computer, research, and 
s kill s for collaborative work and independent study. 

For us, the most important gain from this whole process was that the three groups; graduates, 
employers/supervisors, and graduate directors/professors helped us identify the strengths/weaknesses 
of our academic programs and provided useful recommendations to improve the academic 
preparation of our alumni. This is what makes this whole task a real assessment experience. 
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There are a few things we will improve in our next cycle of the Alumni Study, which will be 
in three years. In terms of the focus groups we thought we would be able to make, at least, two focus 
groups with each class. We managed to organize only one. Although graduates had confirmed their 
attendance, we had a low response for those. This means that we will have to leave more time for 
the organization of the focus groups. 

We will post our Alumni Questionnaire in the web site of our Institution in the Internet. In 
this way, alumni may respond directly to the survey through the Internet. 

The idea of the Employ er/Graduate School Study was good, but it was a real challenge. We 
thought we could use a commercial directory, but we had to make it fi’om scratch. We think that it 
is easier to mail a questionnaire to employers/supervisors of those same graduates. 

Graduate schools on the Island do not gather data about their students. They are concerned 
only with fiilfiUment of admission requirements. We will recommend our programs to follow-up their 
own alumni at graduate level. 

It has been a hard year, but we are convinced that the combination of research techniques such 
as the written survey, the personal interviews, and the focus groups allowed us to overcome the 
challenge of low response rates in our Alumni Study. 

D:\Wilna\PRESENTACION-AIR-VERSION-MAY-24.doc 
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Footnotes 
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^ George Keller (1983). Academic Strategy - The Management Revolution in American 
Higher Education (Baltimore: The Johns Hopkins University Press), Page 129. 



^ See James O. Nichols, (1995). A Practitioner’s Handbook for Institutional Effectiveness 
and Student Outcomes Assessment Implementation (3*^ ed ); Assessment Case Studies: Common 
Issues in Implementation with Various Campus Approaches to Resolution : and The Departmental 
Guide and Record Book for Student Outcomes and Institutional Effectiveness (2nd ed.). Edison, NJ: 
Agathon Press. 



O 

ERIC 



Alumni Surveys 22 



References 



Alexander, J. M., & Stark, J. S. (1994). Focusing on Student Academic Outcomes. In Stark 
& Thomas (Eds.), Assessment & Program Evaluation (pp. 581-584). Massachusetts: Simon & 
Schuster Custom Publishing. 

Astin, A. (1977). Tvyo critical years: Effects of College on Beliefs. Values, and Knovyledge . 
San Francisco: Josey Bass, Inc. 

Banta, Trudy W. (1994). Faculty-Developed Approaches to Assessing General 
Education Outcomes. In Staric & Thomas (Eds.), Assessment & Program Evaluation (pp. 581-584). 
Massachusetts: Simon & Schuster Custom Publishing. 

Doah, H. (1987). Conducting a Graduate Follovy-Up Study . Workshop presented at the 
Twenty-Seventh Annual Forum of the Association for Institutional Research, Kansas City, Missouri. 

Eliza, S. M. & Santos, Jose A. (1990). Estudio de Egresados 1986 . Humacao, PR: UPR- 

CUH. 



Ewell, P.T. (1983). Implementation on Student Oirtcomes: How to get it and how to use 
it. Boulder, CO: National Center for Higher Education Management Systems. 

Ewell, P. T. (1983). Student-Outcomes Questionnaires: An Implementation Handbook. 
Boulder, CO: National Center for Higher Education Management Systems. 

Ewell, P. T. (1984). Conducting Student Retention Studies . Boulder, CO: National Center 
for Higher Education Management Systems. 

Ewell, P. T. & Lisensky, R. P. (1988). Assessing Institutional Effectiveness: Redirecting the 
Self-Study Process. Washington, D.C. : The Consortium in the Advancement of Private Higher 
Education. 

Jacobi, M., Astin A., & Ayala, F., Jr. (1994). College Student Outcomes Assessment: A 
Talent Development Perspective. In Stark & Thomas (Eds.), Assessment & Program Evaluation (pp. 
409-412). Massachusetts: Simon & Schuster Custom Publishing. 

Keller, G. (1983). Academic Strategy: The Management Revolution in American Higher 
Education . Baltimore: The Johns Hopkins University Press. 

Lutz, D. (1994-95). ACT Survey Services Planning Information Booklet . Iowa: American 
College Testing. 




25 



Alumni Surveys 23 



Mid(|le States Association of Colleges and School. (1991). Design for Excellence: Handbook 
for Institutional Self-Study . Philadelphia, PA: Visual Arts Press, Ltd. 

Middle States Association of Colleges and School. (1994). Characteristics of Excellence in 
Higher Education . Philadelphia, PA: Visual Arts Press, Ltd. 

Middle States Association of Colleges and School. (1996). Outcomes Assessment. 
Philadelphia, PA: Visual Arts Press, Ltd. 

Middle States Association of Colleges and School. (1998). Outcomes Assessment Plans . 
Philadelphia, PA: Visual Arts Press, Ltd. 

Nichols, James O. (1995). Assessment Case Studies: Common Issues in Implementation 
with Various Campus Approaches to Resolution . Edison, NJ: Agathon Press. 

Nichols, James O. (1995). The Departmental Guide and Record Book for Student Outcomes 
and Institutional Effectiveness (2nd ed.). Edison, NJ: Agathon Press. 

Nichols, James O. (1995). A Practitioner’s Handbook for Institutional Effectiveness and 
Student Outcomes Assessment Implementation (3*^ ed ). Edison, NJ: Agathon Press. 

Rodriguez, L. R. (1997). Perfil de Egresados del 1995 . Humacao, PR: UPR-CUH. 

Rodriguez, L. R. & Rosario, E. (1998). Perfil de Egresados del 1993 . Humacao, PR: 
UPR-CUH. 

Rodriguez, L. R. & Rosario, E. (1998). Perfil de Egresados del 1996 . Humacao, PR: UPR- 

CUH. 

Rodrigue^ L. R & Rosario, E. (1999). Perfil de Egresados del 1994 . Humacao, PR. UPR- 

CUH. 

Rpdriguez, L. R & Rosario, E. (1999). Perfil de Egresados del 1997 . Humacao, PR: UPR- 

CUH. 



Schreiner, L. A., & Juillerat, S. L. (1994). Student Satisfaction Inventory . USA: 
Noel/Levitz. 

Stark, J.S. (1994). Approaches to Assessing Educational Outcomes. In Stark & Thomas 
(Eds.), Assessment & Program Evaluation (pp. 679-690). Massachusetts: Simon & Schuster 
Custom Publishing. 



Stratil, M. L. (1988). College Student Inventory . Iowa: Noel/Levitz Center. 



Alumni Surveys 24 






Terenzini, P. (1994). Assessment with Open Eyes: Pitfalls in Studying Student Outcomes. 
In Stark & Thomas (Eds ), Assessment & Program Evaluation (pp. 523-536). Massachusetts: 
Simon & Schuster Custom Publishing. 

University of Illinois. (1998). Graduate Survey [On-line]. Available: 

http://www.pb.uiuc.edu/gs/graduate_survey.html 

University of Kentucky. (1994). Alumni Survey . Lexington Campus - Office of Planning 
and Assessment. 




27 




U.5. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 



® 




NOTTCE 

REPRODUCTTON BASIS 




This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, m.ay 
be reproduced by ERIC without a signed Reproduction Release form 
(either “Specific Document” or “Blanket”). 




EFF-089 (9/97) 




